So it turns out there's actually very little public domain music available -- at least in the USA. Even those 100 year old scritchy-scratchy recordings aren't in the public domain. Here's an explanation why:
http://www.publicdomainsherpa.com/public-domain-sound-recordings.html
The author of this page has an excellent list of public domain recordings here:
http://www.publicdomainsherpa.com/public-domain-recordings.html
The few samples of public domain music & spoken word you see in this folder were largely retrieved from:
PUBLIC DOMAIN MUSIC:
http://en.wikipedia.org/wiki/Wikipedia:Sound/list
http://www.gutenberg.org/browse/categories/3
http://musopen.org/
http://us.metamath.org/mpegif/mmmusic.html
http://www.archive.org/details/audio
http://memory.loc.gov/ammem/browse/ListSome.php?format=Sound+Recording
PUBLIC DOMAIN SPOKEN WORD:
The George W. Bush Public Domain Audio Archive